Virtual Machine Security Systems

Current operating systems provide the process abstraction to achieve resource sharing and isolation. From a security perspective, however, an attacker who has compromised one process can usually gain control of the entire machine. This makes security systems running on the same computer, such as anti-virus programs or intrusion detection systems, also vulnerable to attack. In response to the imperfect isolation between processes in modern operating systems, security researchers have begun to use virtual machine technology when designing security systems. A virtual machine makes raw device requests to a set of devices that are emulated by underlying software. So, software running in a VM has the appearance of its own dedicated hardware, but is actually controlled and managed by a software layer on the physical computer. With reasonable assumptions, the level of isolation between virtual machines is nearly equivalent to having two separate machines. At the same time, the underlying software has full access to each virtual machine’s state. These properties make virtual machines very attractive for designers of security systems. In this chapter we explore a wide variety of security applications that utilize virtual machine technology, including intrusion detection and introspection, honeyfarms, logging and replaying, secure file systems, and even malicious software.